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Purpose 


U.S.  AIR  FORCE 


■  THUNDER  vs.  STORM 

■  Goal  of  DOE  here:  V&V  by  comprehensive  comparison 

■  Why  not  VV&A? 

■  General  and  specific  models  of  the  V&V  experiment 

■  DOE  for  adequate  excitation  to  compare  responses 

■  Screening  to  reduce  response  model  size 

■  Adding  design  points  to  check  for  interactions 

■  Comparing  THUNDER  and  STORM  response 

■  By  Singular  Value  Decomposition 

■  By  Canonical  Correlation 

■  Automating  execution  of  runs  and  extraction  of  results 

■  Work  in  progress  -  all  feedback  welcome 

■  Punch  line:  Massive  quantitative  validation  experiment  -  uses 
computers  and  sophisticated  techniques  to  exceed  past  human 
limits  on  data  production  and  interpretation 


UNCLASSIFIED  I  integrity  -  Service  -  Excellence 


U.S.  AIR  FORCE 


UNCLASSIFIED 

THUNDER  vs.  STORM 


■  THUNDER:  accepted  standard  ■ 

■  Not  Government  owned 

■  Proprietary  engine, 

SIMSCRIPT 

■  CACI  licenses  - 
big  $$$  /  seat  /  year 

■  Ground  war: 

■  “Pistons”  define  FLOT 

■  Piston  walls  do  not  reorient  as 
FLOT  distorts 

■  No  study  manager  --  manual 
file  configuration  control 

■  Minimal  GUI 

■  Minimal  ISR 


STORM:  AFSAA’s  next  standard 

■  Government  owned 

■  Open  source,  non-proprietary 

■  Simulation  engine  in  g++  ; 
maps,  DB  tools;  even  OS,  all  open 

■  No  per-seat  license  fees 

■  Ground  war: 

■  Along  arcs  and  within  nodes 

■  Shape  of  FLOT  morphs  naturally 
under  network  constraints 

■  Study  manager  automates 
file  configuration  control 

■  Extensive  GUI  -  map  tool, 
report  tool,  graph  tool 

■  Sat  tool  and  other  significant 
ISR  developments 
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Y  v&  V  by  Comparison  (Not  W&A  ?) 

U.S.  AIR  FORCE 


■  v&v 

■  Verification:  “Solve  the  Problem  Right” 

■  Identify  and  fix  lurking  code  problems 

■  Validation:  “Solve  the  Right  Problem” 

■  Accepted  referent  is  THUNDER  but  comparison  is  a  hard  problem 

■  Comparison  needed  over  a  set  of  scenarios  of  interest 

■  Are  results  “comparable  to  or  better  than”  THUNDER?  Just  what 
does  that  mean? 

■  SME’s  “face  validation”  vs.  statistical  tools  -  both  are  needed 

■  Does  DOE  show  effects  of  changes  in  STORM  vs.  THUNDER? 

■  Why  V&V,  not  W&A? 

■  A  ( Accreditation  )  is  PM’s  decision  -  balancing  risk 
versus  resources 

■  Too  little  V&V:  low  resource  expenditure  but  high  risk 

■  Too  much  V&V:  little  risk  reduction  for  last  expenditures 
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A  General  Model  of  an  Experiment 

U.S.  AIR  FORCE 


Prior  =  before  running 
the  experiment  or 
getting  the  data 


Prior 
“knowledge” 
or  Belief 


Where  and  how 
one  samples 
Nature’s  response 


Response 

Sampling 


What  one  offers 
up  to  Nature  to 
excite  a  response 


Posterior 
“knowledge” 
or  Belief 


Posterior  =  after 
getting  the  data 
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U.S.  AIR  FORCE 


UNCLASSIFIED 

...Tailored  to  V&V and  Comparison 


STORM  uses  next 
generation  THUNDER 
campaign  model 


Prior 

knowledge 
or  Belief 


Response  Sampling  is 
MOMs  +  dots  extracted 
from  simulation: 
the  OUTPUTS 


Response 

Sampling 


Excitation  is  the  set  of 
scenarios,  each  with  its 
experimental  design  and 
with  levels  for  factors: 
the  INPUTS 


Posterior 
knowledge 
or  Belief 


STORM  improves  on 
THUNDER -it’s  a 
good  replacement 
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V&V  Scenario  Set  of  Interest 


■  Consult  with  SMEs,  see  for  what  campaign 
models/scenarios  THUNDER  results  (inputs  and 
outputs)  are  already  available  and  of  interest 

■  THUNDER  has  been  much  more  expensive  to  run 
(small  error  =>  typically  60-100  reps)  than  STORM 
-  hence,  minimize  additional  THUNDER  runs 

■  Choose  among  available  THUNDER  campaigns  to 
agree  on  sufficiently  complete  V&V  scenario  set 

■  If  necessary,  add  THUNDER  or  STORM  work  to 
achieve  common  and  complete  set  of  scenarios  of 
significant  interest  for  V&V 
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Need  to  Reduce  Input  Dimensions 


U.S.  AIR  FORCE 


■ 

THUNDER 

STORM 

v6.9 

vl.3 

99 

115 

*.dat  files 

.dat  fileS  {plus  path.dat) 

■  Typical  .dat  file  has  10  to  1000+  parameters  (inputs)  -  highly 
interrelated,  often  nonlinearly 

■  Too  many  parameters  to  screen  individually 

■  Use  SME’s  to  combine  (cull?)  inputs  into  fewer  (10  to  100) 
factors  fj  =def  [  Pj j.  ]  parameter  vectors 

■  Pj  j ,  Pj  k9tj  need  not  be  in  same  .dat  file 

■  Each  parameter  in  a  factor  has  two  levels,  A  /  B  -  goes  A  /  B  as 
its  factor  goes  high  /  low  -  no  “within-factor  interactions” 

■  Factor  screening  takes  place  in  STORM  across  entire  V&V  scenario 
set  at  once 

■  Whether  the  change  in  level  of  factor  f;  is  significant  depends  both  on 
the  scenario  set  and  on  what  MOM(s)  or  output  dof(s)  are  of  interest 
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Input  Sensitivity  Screening 


■  Initial  screening  in  STORM  of  two-level  factors 

■  Resolution  V  (preferred)  or  IV  (if  necessary) 
fractional  factorial  (FF)  -  possibly  even  res.  Ill  for 
initial  stab? 

■  If  resolution  IV  necessary,  follow  up  with  resolution  V  (res.  IV  significant 
factors  only,  plus  all  2-factor  interactions)  to  screen  for  significant 
interactions 


■  256  reps  (  resolution  IV )  =>  up  to  128  factors 

■  Comparable  to  #  of  reps  in  STORM  V&V  reports  for  previous  releases  ( 

19  +  cases  x  10  repeated  reps  /  case  ) 

■  Likely  fewer  factors  (<64,  <32)  if  long  reps  (tsim»10  days) 

■  However,  256  reps  (  resolution  V  )  => 

only  12  to  17  factors  (  +  all  2-level  interactions  ) 

In  any  case  - 

■  NOT  One-Factor-At-a-Time  as  before! 

■  NO  replicated  design  points  (reps)!  ——  ■  ,  , 

Stmu 
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Matrix  Model  Equations 


Y=  XB  +  E 


/y  /v  /v 


Y  =  XB  +  E(s) 


B  is  by  least  squares,  minimizes 


rss  =  2;  (y.. 
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Not  Your  Usual  Few  Responses 

U.S.  AIR  FORCE 


m  =  #  of  response  variables 
d  =  #  of  design  variables  ( factors  ) 
n  =  #  of  runs 
n  >  d  (always) 
m  «  d  (usually) 


< 


m  »  d 

- ► 


Why  does  Y  need  to  In  our  V&V  experiment,  X  is  excitation;  Y  samples 
be  so  broad,  m  »  d  ?  response.  We  are  deliberately  oversampling. 


We  oversample  by  several  times  the  dimensionality  of  our  excitation  in  a  “shotgun 
approach”,  hoping  we  catch  most  of  the  significant  dimensions  of  model  response. 
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V  Why  Fit  Is  Always  Too  Good 

U.S.  AIR  FORCE 


B 


U.S.  AIR  FORCE 
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Elements  of  Matrix  Model 


■  Starts  as  typical  multivariate  response  model 

■  Response  matrix 

■  Design  matrix 

■  Effects  matrix 

■  “Out-of-model”  matrix 

■  Estimates  “hatted”: 

■  Beyond  typical  model:  £ 

■  Compute  Y  from  X  in  permuted  order 

pco^\  pco&\  pco&\  but  record  Y  in 
design  order  (picture  of  Y ,pcoi,pco^  later) 

■  Record  random  seed  and  permuted 
computational  order  [  s{  pcot\  for  row  i 


(pxm) 

X 

(Rxm) 

E 

/V 

B 


(nxd) 

(dxm) 


(nx2) 
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UNCLASSIFIED 

Design  Matrix  X  -  Words 


■  ROWS  in  the  design  matrix  X 

■  Give  particular  combination  of  factor  levels  for  each 
design  point 

■  Cover  all  scenarios  of  interest 

■  Contain  a  non-zero  entry  in  only  one  of  the  columns 
indicating  scenarios 

■COLUMNS  in  the  design  matrix  X 

■  Signify  design  (input)  factors 

■  Signify  scenarios  for  estimating  mean 

■  Decompose  sum  of  squares  due  to  scenario  means 

■  Number  of  means  is  included  in  d,  #  of  design  variables 
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Design  Matrix  -  Picture 


1 ' s  account  for 
scenario  means,  ssq 
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9  10  factor  level 


constant  factor  level 
within  scenario  says 
significance  of 
factor  is  not  tested 
within  scenario 


Notional  design 
only,  not  2  r-p 
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Response  Matrix  Y  -  Words 


■  ROWS  in  the  response  matrix  Y 

■  Correspond  to  design  matrix  rows 

■COLUMNS  in  the  response  matrix  Y  signify 

■  Scalar  or  vector  MOMs  /  dofs 

■  Consistent  responses  for  each  row  and  scenario 

■  Squadrons  or  other  entity  types  handled  differently  by 
planning  files  on  different  design  points  should  not  re-use 
the  same  response  columns  unless 

■  the  difference  in  response  due  to  difference  in  planning  file  parameters  is  the 
effect  being  sought  and 

■  the  difference  in  planning  file  parameters  is  reflected  in  one  or  more  factors 
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Response  Matrix  -  Picture 


pco  ( ' ) 

pco  j 

scenario  # 

i  =  row  #  in  X 

Scalar  MOM  1 

Scalar  MOM  2 

Scalar  MOM  3 

Vector  dof  1  day  1 

Vector  dof  1  day  2 

Vector  dof  1  day  3 

Vector  dof  1  day  n-1 

Vector  dof  1  day  n 

Scalar  MOM  4 

Vector  dof  2  day  1 

Vector  dof  2  day  2 

Vector  dof  2  day  3 

Vector  dof  2  day  n-1 

Vector  dof  2  day  n 

Scalar  MOM  5 

Vector  dof  3  day  1 

Vector  dof  3  day  2 

; 

Vector  dof  3  day  n-1 

Vector  dof  3  day  n 

Vector  dof  4  day  1 

etc. 
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Effects  Matrix  B,  B  hat 


■  ROWS  in  the  effects  matrix  B 

■  Correspond  to  design  variables 

■  For  SME-aggregated  factors,  coefficient 
shows  effect  of  change  in  level 

■  For  scenario  indicator  variables,  coefficient 
shows  scenario  mean  for  each  response 

■COLUMNS  in  the  effects  matrix  B 

■  Correspond  to  response  MOMs,  dofs 

ulu 
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Out-of-Model  Matrix  E,  E  hat 


■  Part  of  E  is  assumed  due  to  stochastic  effects 

■  Neither  random  seed  value  5,  nor  its  effect  on 
each  response^  is  known  to  the  fitting  process 

■  ,v;-  and  fj(*)  —  hence  —  are  termed  exogenous 

(excluded  from  model) 

■  Effects  of  exogenous  variables,  including  stochastic 
effects,  alias  to  an  unknown  extent  onto  endogenous 
(included  in  model)  effects  bj 


^  1 
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Interactions ,  Response  Surface 

U.S.  AIR  FORCE 


■  Once  factors  and  2-factor  interactions  are  identified 
as  significant  at  resolution  V,  high-low  level  only  ... 

■  Add  intermediate  levels  as  appropriate  to  gain 
Response  Surface  Model  (RSM) 

■  Central  composite  design  if  appropriate 

■  Retain  only  statistically  significant  factors  (and 
interactions)  for  comparing  STORM  to  THUNDER 

■  Not  normal  statistical  practice  to  “cherry  pick” 
interactions  without  physical  model  justification 

■  Empirically  justifiable  because  RSM  is  descriptive 
only,  not  first  principles 

■  We  are  a  long  way  off  from  this 
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Dimension  Reduction  (1):  RSM 

U.S.  AIR  FORCE 


■Add  face  and  center  points  for  significant 
factors  to  get  central  composite  design  if 
feasible 

■  Possibly  use  intermediate  points  for  near¬ 
orthonormal  Latin  Square 

■Get  response  surface  model  (RSM)  - 
polynomial  or  spline 

■  Stop  on  Mallow’s  Cp  or  other  measure  of 
complexity  vs.  goodness  of  fit 

■  Retains  coordinates  of  original  model  space 
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Weight  Questions  (1) 


■Question  1:  What  is  the  proper  weight  or 
scale  per  MOM  /  dof(s)  in  the  penalty 
function  for  lack  of  fit? 

■  That  is,  how  much  should  lack  of  fit  in  one  column  (or 
set  of  columns)  of  Y  count  relative  to  lack  of  fit  in 
another  column  (or  set  of  columns)? 

■Question  2:  Same  as  Question  1  but  for 
one  scenario  (set  of  rows)  versus  another 
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Weight  Answers  (Partial)  (2) 


■  Depends  on  user  goal  and  on  STORM  / 
THUNDER  comparison  approach 

■  Data  for  SVD,  canonical  correlation  should  have 
zero  scenario  mean 

■  Appears  no  natural  scale  for  Singular  Value 
Decomposition  (SVD);  use  unit  column  variance 
for  consistency  with  canonical  correlation 

■  Scale  set  to  unit  column  variance  by  definition  of 
canonical  correlation,  but  meaning  and 
appropriateness  to  user  goal  are  not  clear 

■  Beware  augury  /  mysticism  /  wise  chin-scratching 
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Equations  in  Passing 


■  Final  dry  run  for  this  brief  commented  that  there 
were  way  too  many  equations,  too  few  graphics 

■  Graphics  proved  far  more  time  consuming  to 
produce  than  initially  expected 

■  Equations  were  retained  more  as  art  objects  and 
speaking  totems  than  as  detailed  communication 
tools 

■  Equations  past  the  basic  “Matrix  Model  Equations” 
will  not  be  on  the  midterm  or  the  final 
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Dim.  Red.  (2):  SVD  (1) 


■Singular  Value  Decomposition  (SVD)  on 
responses  Y  adjusted  to  zero-scenario-mean, 
scaled  to  unit  Y  column  norm,  and  transposed*: 


'  not  a  good  idea,  but  not  fatal 


Y 

zsm 
( nxm ) 

A 

( mxn ) 


YfT  —X  (X  'X  V!X 

\n  ^^sm  \  Vvw  sm  ) 


"V 


X  vm  =  only  columns  for  scenario  means 


iy  r 

zsm  J  5 


j 


A  =  ODV'  \ 


(y  )=(y  )../,/ Y”  (y  j. 

\S  zsm  Hj  \S  zsm  Hj  /  V  /  jj'=\  V-7  zsm  h\ 

'0'0  =  I„,(m>«)=>00VI„ 

D  diagonal,  p.s.d.,  (/  <  /)  =>  (du  >  d..) 
VV'=  V'V  =  I_ 


V 
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...  SVD  (2) 


w 
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■Continuing, 

A’ A  =  Z  =  VD’O'ODV’ 

=  VD2V’ 

ZV  =  VD2V'V  =  VD2 

ZV  =  VA  (symmetric  eigenproblem) 

du  =  sfc  (sorted  Al>A2>...An>  0) 

O ,  =  A y,d..~  canonical  column  i  of  A  for  dn  >  0 

L  L  L  L  1/  L 

■Why  is  this  significant? 
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...  SVD  (3) 
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■  This  is  significant  because... 

A  =  ODV'=  Td„OV  (exactly!)  and 


ZZ(01.V1.%  =  ZZ(o;,vjt)2=ZoJZvr 

J  k  J  k  J  k 


ik 


so  all  scale  information  is  in  the  “singular  values”  du  in  the  sense 
that  rf;;2  shows  how  many  unit  rows  of  A  are  expressed  by 


■  This  means  that  the  completeness  R2(k,m)  of  expressing  all 
m  unit  rows  of  the  response  matrix  A  using  only  k  coefficients  is 


■  More  on  SVD  follows  after  discussion  on  Canonical  Correlation 
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DR  (3):  Canonical  Correlation  (1) 


■Two  response  sets,  say  T=Y’t  for  THUNDER 
and  S=Y’s  for  STORM 

mm  ROWS  of  T,  S 

■  Zero  scenario  mean,  unit  variance  as  for  SVD 

■  Same  row  <^>  same  MOM  or  dof  for  T  as  for  S 

mn  COLUMNS  of  T,  S  are  in  same  order  as 
design  matrix  rows,  although  design  points 
were  computed  in  permuted  sequence(s) 
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CanCorr  (2) 
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■  Canonical  correlation  (CanCorr)  gives  mx 2  coefficients  [  lea,  leb  ] 
for  successive  2  linear  composites  [  wj,  such  that  for 

11'=  Ica  'T,  w  '=lcb  'S,  k  =  1,  2, 111  in( /;/./;) 

■  11.  and  w^have  maximal  correlation  p*L  between  any  two 
linear  composites  of  T,  S  not  spanned  by  [Uj  u2  ...  uM]',  [Wj  w2 
...  wM]' 

■  Var(u  k)  =  Var(uk)  =  1 

■  P*i  —  P*2  —  P*3  —  •  •  •  P*min(m,n)  —  0 

■  For  j>k,  i^andw^are  uncorrelated,  i.e., 


U/(I-1(1'1)-11')  w*  =0 
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...  CanCorr  (3) 


How  does  this  work?  From  the  zero  scenario  means, 
unit  row  norms,  and  definition  of  variance  we  get 


0 

(2mx\) 


T 

S 


1 

(ftxl) 


2  = - - 

(2mx2m)  n~f l 


zsm 


T 

T]' 

S 

s 

12 


f  y  y  \  f  v  v  A 

^ts 

V^ST  ^SS  J 


V^21  ^22  J 
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CanCorr  (4) 
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Then  it  can  be  shown  in  about  3  textbook  pages  that 


u 


k 


W 


k 


yy-H2  rp 

e k  ^11  1 


— v 
lea. 


f  »y-l/2n 

^22  a 

' - V - ' 

leb  l 


,  where 


,  1 

Confu^w*)  =  pk  is  the  k  1  largest  eigenvalue  of 
E'1/22:i2E“‘2:2iE“i1/2,with  eigenvector  ek,  and  also  of 
'1/2L.-,Lr!L^L2/2,  with  eigenvector  f , . 
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...  CanCorr  (5) 


■  First  approach  will  be  to  look  at  initial  canonical 
variables  (cv’s)  as  long  as  pk  are  well  separated 

■  If  pk  are  closely  spaced,  eigenvectors  and  hence  cv’s  are 
determined  only  up  to  rotations 

■  Second  approach  will  be  to  look  at  residuals  once 
projections  on  all  cv’s  from  1  to  k  are  removed,  for  k’s 
such  that 

■  k  is  not  in  the  middle  of  a  closely  spaced  block  of  pk 

■  k  is  not  so  far  beyond  1  that  any  patterns  apparent  in 
residuals  are  as  likely  to  be  artifacts  of  numerical  methods  as 
of  remaining  THUNDER  vs.  STORM  relationships 


■  We  have  not  done  this  before;  any  advice  would  be 

welcome. 
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SVD  versus  CanCorr ;  SVD  (4) 
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■  A’ A  =  Z  of  the  SVD  development  can  now  be  seen 
as  Z22  of  CanCorr. 

■We  also  have  E12,  the  covariation  of  T  with  S.  We 
can  estimate  what  we  would  expect  for  T  based 
on  S  using  the  first  k  singular  vectors  of  S  as 


T(S,  £)  =  L12(L22)_1|i  =  £120 


O 


0 


V 
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SVD  versus  CanCorr:  SVD  (5) 
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■Similar  remarks  and  approaches  as  for  CanCorr 
apply  to  SVD 

■  Singular  vectors  are  less  well  defined  as 
eigenvalues  become  closely  spaced 

■  We  may  look  at  the  fits  (estimated  values)  first  if 
most  of  the  significance  falls  into  a  few  numerically 
stable  patterns 

■  We  will  look  at  residuals  as  well,  particularly  at 
breaks  in  a  block-type  pattern  of  decrease  in 
singular  values  /  eigenvalues 
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Y  SVD  versus  CanCorr:  SVD  (6) 
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■The  big,  big  difference  between  SVD  and 
CanCorr  is 

■  SVD  aims  to  compactly  express  the  variation  of  a 
single  data  set 

■  Covariation  with  the  other  data  set  is  dealt  with 
almost  as  an  afterthought. 

■CanCorr  aims  to  compactly  express  the 
covariation  of  both  data  sets,  without  favor  for 
one  or  the  other. 
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Automating  Runs  and  Results  (1) 


■  Again,  a  work  in  progress  -  all  feedback  welcome 

■  Number  of  parameters,  factors,  runs  makes  manual 
DOE  setup  and  execution  unattractive  for  all  but 
small  problems 

■  Manual  "baby  steps”  on  small  problems  for  debugging 

■  Concept  is  to  factor  problem  into  DOE  files  giving 

■  Design  matrix,  source  citation,  permuted 
computational  order 

■  Factors  and  associated  parameters,  with  .DAT  file 
levels  for  parameters  as  .DOE  factor  goes  between  its 
levels 
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Automating  Runs  and  Results  (2) 


■  Automate  run  process 

■  Substituting  parameter  .DAT  levels  into  .DOE  files 

■  Write  result  to  temporary  .DAT  files 

■  Execute  runs  in  permuted  order 

■  Automate  DOE  post  process 

■  Extract  results 

■  Transfer  to  statistical  software 

■  Use  stat  tools  for  evaluation  and  comparison  between 
THUNDER,  STORM  results  on  nominally  equivalent 
models 
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Punch  Line:  Quantitative  V&V? 
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■  This  is  a  massive  quantitative  validation  experiment 

■  Uses  computers  to  exceed  past  human  limits  on  data 
production 

■  Uses  computers  and  sophisticated  math  to  surpass 
unaided  human  limits  on  data  interpretation 

■  Unexpected  differences  between  THUNDER  and 
STORM  will  constitute  new  knowledge 

■  Finding  only  differences  explainable  as  due  to  known 
deliberate  changes  would  be  a  disappointment 
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Nothing  to  Lose  But  Uncertainty 


■  The  author’s  involvement  stems  from  >  20  years  trying  to 
compare  test  data  with  modeling  results 

■  16  years  shock  testing  nuclear  powered  ships  and  their 
components 

■  9  years  with  robotic  test  equipment  testing  composite 
material  systems 

■  Assumed  material  response  theory  (dissipated  energy  density)  said 
could  account  for  60K  data  points  via  125  coefficients  and  constraints 

■  Ability  to  so  compactly  model  test  data  gave  massive  support  to 
underlying  theory 

■  It  is  long  past  time  for  a  similar  quantitative  approach  to  V&V  - 
and  comparing  THUNDER  and  STORM  should  be  a  milestone  in 
that  effort. 
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Questions? 


UNCLASSIFIED 


Integrity  -  Service  -  Excellence 


